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Quality assurance and e-learning: blue 
skies and pragmatism 
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This paper considers the role of quality assurance in e-learning; reflecting on the conditions neces- 
sary for successful e-learning. It reviews some of the current international work on quality assurance 
in this area and goes on to consider the ways in which the quality of a process or activity can be 
assessed — focusing on the use of benchmarking and specification of standards. 


Introduction 

There is still a high degree of scepticism and concern being expressed by many that 
the use of information and communications technology (ICT) in education remains 
an unproven experiment, despite the huge investments in technology and infrastruc- 
ture and the high levels of interest that remains among many educators, administra- 
tors and policy-makers (for example, Cuban, 2003; Oppenheimer, 2003). The 
principal concern expressed by many critics of ICT in education is the lack of empir- 
ical evidence for learning enhancement, and ICT is often seen to do more harm than 
the good it generates as a consequence of the overheads and associated costs (for 
example, Oppenheimer, 2003). In higher education, where ICT in teaching and 
learning is probably more prominent than in any other sector, this degree of scepti- 
cism appears just as strong (for example, Romizowski, 2005). Large-scale moves to 
use technology to support teaching and learning have been seen to be premised on 
poor assumptions and inaccurate perceptions of public response. 

Many projects such as the UK eUniversity, NYU Online, Scottish Knowledge, 
Universitas 2 1 and Global University Alliance, which all developed around e-learning 
applications, have failed to realise their aims and goals, leading many to question the 
quality and capabilities of this form of educational delivery (Garrett, 2004) . Like all 
forms of education, there are both good and bad examples in practical settings. The 
questions many people are looking to answer are: what are the necessary conditions 
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for successful e-learning, and can these conditions guarantee that e-learning will be 
successful? Many of these questions have become more important in the current era 
where accountability is a key concern in the higher education sector. 


The quality agenda 

There is currently a quality agenda running among educational institutions world- 
wide, and within higher education in particular. It is a prominent and mainstream 
activity that seeks to ensure that there is accountability in the ways in which institu- 
tions go about their daily work. The notion of quality in the delivery of education is 
generally contextualised within three possible definitions: quality as value for money, 
quality as fit for purpose of the institution or quality as transforming (Biggs, 2001). 

Biggs (2001) argues that the quality agenda that confronts many in higher educa- 
tion today involves mainly quality assurance processes based on public accountabil- 
ity. For example: 

• accountability to a funding body, for example, the Government; 

• a desire to improve outcomes; 

• the prospect of new opportunities; 

• being able to sustain programmes and activities; and 

• an ability to demonstrate achievement against stated goals. 

Biggs calls such quality assurance processes retrospective activities, because they look 
back to see what has been done rather than looking forward {prospective ) to see what 
can be done to transform and change educational processes to improve the service 
delivery. 

The movement, while worldwide, appears to have grown mainly from activities in 
the United Kingdom where quality assurance processes have been in place for some 
time. The Quality Assurance Agency for Higher Education (QAAHE) in the United 
Kingdom describes the aims of quality assurance of teaching and learning in higher 
education as being to: 

• contribute, in conjunction with other mechanisms, to the promotion of high qual- 
ity and standards in teaching and learning; 

• provide students, employers and others with reliable and consistent information 
about quality and standards at each higher education institution (HEI); 

• ensure that HE programmes are identified where quality or standards are unsatis- 
factory, as a basis for ensuring rapid action to improve them; provide one means of 
securing accountability for the use of public funds received by HEIs (QAAHE, 

2001, p. 2). 

There is a similar high degree of quality activity in higher education in Australia as a 
consequence of the actions of the Australian University Quality Agency. This agency, 
acting under the authority of the Commonwealth Government, conducts regular 
quality audits and provides public reports on the quality assurance arrangements of 
self-accrediting higher education institutions in Australia. This quality audit process 
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encompasses all aspects of institutions’ activities including, teaching and learning, 
research and management, including those activities that involve offshore elements. 

There is an abundance of literature that describes quality in teaching and learning 
in higher education (for example, Ramsden, 1992; Laurillard, 1993). Quality assur- 
ance in this setting is achieved primarily through detailed examinations of the broad 
aims and goals across such diverse aspects of the teaching programmes as the curric- 
ulum offered, its implementation, assessment processes and student learning 
outcomes. Questions that are fundamental to underpinning an examination of an 
organisation’s teaching and learning practices and outcomes include the following 
(University of Tasmania, 2000): 

• What quality assurance policies and practices does the institutions have in place or 
in the process of development to assure the quality of its teaching and learning 
performance? 

• How effective and how fully deployed are these? 

• What processes does the institution have to evaluate and monitor the quality of its 
outcomes? 

• What quality-related indicators does the institution use and why? 

• What are the institution’s priorities for improvement? 

• What quality initiatives has the institutions undertaken (since the last review) and 
what evidence of improved performance is there? 

As technology has become increasingly applied in education, its use in the delivery 
of teaching and learning has led many to distinguish this as a discrete educational form, 
called e-learning (for example, Stella & Gnanam, 2004). As a teaching and learning 
activity, e-learning has now been caught up in the quality agenda. Many national 
bodies and organisations have now developed principles, guidelines and benchmarks 
to describe quality in the use of technologies to support flexible learning. For example, 
the Institute for Higher Education Policy (2000) in the USA describes a series of 
benchmarks that are argued as essential to ensuring excellence in Internet-based 
distance learning. Likewise, in the United Kingdom the Quality Assurance Agency 
(QAA) describes a code of practice for the assurance of academic quality and standards 
in the provision of flexible and distributed learning, including e-learning (QAA, 2004). 

E-learning is a form of educational delivery that has become quite prominent in 
universities worldwide and an activity that, to all intents and purposes, can now be 
considered mainstream. Despite its uptake, however, as with conventional teaching 
in higher education, e-learning is often done well and poorly in different locations 
within the same university. One of the main problems with e-learning is that its prac- 
tice has evolved from conventional teaching and learning, and its application tends to 
involve variations and copies of face-to-face teaching rather than practices developed 
from a ‘green-field’ approach (for example, Collis & Moonen, 2001). Contemporary 
research is continuing to explore e-learning as a discrete activity and to distil factors 
that influence its success and achievements. There is still uncertainty and doubt 
among many as to what actually constitutes a quality e-learning approach, and it is 
this question that this paper seeks to explore. 
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Quality and e-learning 

One of the problems facing those who seek to describe quality in e-learning is under- 
standing precisely what constitutes e-learning. E-learning occurs in a wide range of 
teaching activities where technology of one form or another is involved. Technology 
necessarily underpins the administrative functions of most universities and higher 
education institutions and, for many, the lines between the administration, and the 
conduct, of teaching can be blurred. E-learning takes many forms, and common 
instantiations in e-learning delivery and approaches include: 

• flexible learning, technology supports for learning any time, anywhere; 

• blended learning, varying mixes of technology with conventional learning usually 
in conventional settings; and 

• online learning, where technology provides the means for the implementation and 
delivery of learning programmes totally distinct from face-to-face teaching (for 
example, Fresen, 2005). 

Within this diverse and broad range of activities, we are now seeing increased levels 
of awareness and concern for the quality of the activities that result. There is a height- 
ened level of interest in being able to monitor and review performance and to demon- 
strate successful outcomes. In catering for the diversity, most exercises in quality 
assurance steer towards the activities with the highest levels of technology use and 
dependence; for example, distance education and online learning (for example, Insti- 
tute for Higher Education Policy, 2000; QAA, 2004). Once parameters have been set 
for excellence and best practice in this form, the variations that exist among e-learning 
examples that come from a blended learning or flexible delivery form can still be 
adequately accommodated. 

The literature that describes successful teaching and learning in online settings 
typically draws from research that has focused on discrete aspects of e-learning in 
particular settings. For example, a research project might explore levels of learner 
participation in an online discussion. The myriad of research clearly shows that there 
is a large range of activities and outcomes upon which the successful application of e- 
learning processes depends. In the big picture, these include such aspects as: 

• the scope and nature of the learning materials; 

• appropriate selection of the learning design; 

• the levels of learner engagement; 

• extent of community development within virtual settings; 

• scope and level of flexibility of learning; 

• enhanced learning; 

• reusability of the resources; 

• the accessibility of the resources; and 

• the level of uptake of e-learning among staff (for example, Khan, 1997; Kearsley, 
2005). 

In an era of increased accountability, it is important for stakeholders in educational 
organisations to be able to demonstrate that their approaches to e-learning are sound 
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and effective. They need to be able to demonstrate that they are carry out e-learning 
in a way that is efficient and productive, and they must be able to demonstrate quality 
in the curriculum, the delivery and their teaching and learning approaches. 


Assessing quality 

There are usually two main ways by which the quality of a process or activity can be 
assessed, through benchmarking or by the specification of standards. Benchmarking 
compares the performance and outcomes in one setting against that achieved by 
selected others operating in a similar sphere, a process of relativity; whereas the use 
of standards uses criterion-related references to judge performance. Benchmarking 
enables an organisation to see how its performance compares against others. It can 
demonstrate what is good or poor practice through comparisons against established 
best practice. It is a process commonly used in industry in which businesses use 
known leaders as models and targets. The process involves comparing local practices 
against known best practice to determine where there is a need to improve. In indus- 
try, benchmarking systematically uses quality assurance processes to identify exam- 
ples of excellence and best practice, and then uses these examples as the standard of 
comparison (for example, Achtemeir & Simpson, 2005). 

Benchmarking is a difficult process to apply in most university settings. McKinnon 
et al. argue that: 

No area of university life [learning and teaching] is more difficult to benchmark. It is char- 
acteristic of universities that the courses and the approach to teaching them are not stan- 
dard. Courses, even professional courses leading to registration, are rarely, readily or 
directly comparable. There will always be diversity. (2000, p. 69) 

E-learning is one activity in university settings where benchmarking processes might 
be employed to ascertain quality. E-learning comprises discrete and distinct teaching 
and learning elements that can be isolated and identified for benchmarking purposes. 
There currently exist a number of standards and guidelines that have been developed 
to aid this process (for example, Institute for Higher Education Policy, 2000; Learn- 
ing Object Metadata (LOM), 2002; Learning Technology Standards Committee 
(LTSC), 2002; QAA, 2004). At the same time, there has been considerable research 
and development in e-learning and many reports have resulted that showcase exam- 
ples of best practice and that can be used as potential benchmarks against which 
comparisons can be drawn. The difficult aspect in this process is perhaps the steps 
associated with making judgements concerning the choice of best practice examples, 
which might be used as a benchmark example. 

A number of researchers have attempted to provide frameworks that can be used 
to provide overarching models to describe the critical elements of learning settings 
that can be used to contextualise the factors influencing effective outcomes (for exam- 
ple, Sims et al., 2002). Typically the frameworks distinguish four discrete elements; 

• The curriculum , that which is to be learned. A strong curriculum has relevance to 
the student and the workplace. It has currency and reflects best practice. 
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• The learning design, the planned learning environment. An effective learning design 
provides the forms of learner engagement required to assist the learner to interact 
with that which has to be learned in meaningful ways. 

• The learning resources, the course content. Strong course content is accessible and 
current. It provides multiple perspectives and conceptual underpinning. 

• The delivery processes, supports and scaffolds for learning. A strong delivery process 
supports the learners, provides contexts for communication and collaboration. 

Within these four elements, there are examples of best practice that apply to teaching 
and learning in general, and examples that could be considered unique to e-learning. 
Since e-learning is primarily a descriptor of the medium of instruction, descriptions 
of best practice would tend to apply mainly to instantiations of a curriculum and 
course, more than the design of the course itself. From the earlier list, this leaves 
learning designs, learning resources and delivery processes as the elements of e- 
learning that could form the basis of any benchmarking activity. 


Learning designs for quality e-learning 

The term learning design (see for example Britain, 2004) describes the deliberate 
learning activities and processes that a teacher or instructional designer plans into a 
learning environment to provide the cognitive engagement a learner is deemed to 
require in a learning experience, to bring about the required conceptual change asso- 
ciated with the planned learning outcomes (Kalantzis & Cope, 2004). Learning 
designs may be at the level of a whole subject, subject component or learning resource 
(for example, Hedberg et al., 2002). Boud and Prosser (2002) argue that high-quality 
learning activities must demonstrate four principles: 

• Engagement of learners. 

• Acknowledgement of context. 

• Challenge for learners. 

• The involvement of practice. 

E-learning settings across all sectors of education have long been criticised for their 
limited and shallow learning designs (for example, Mioduser et al., 1999). Typical 
online courses are usually comprised of comprehensive electronic resource sets and 
information with little intentional instructional design aimed at supporting meaning- 
ful learning. The most common forms of learning design involve students reading 
screen-based texts and answering questions designed to promote engagement. 

In a large study of technology-based learning examples undertaken in Australia 
in 2003, a number of different learning designs supporting quality learning experi- 
ences were identified and described and exemplars included into an online data- 
base (Australian University Standards Committee, 2003). The database was 
designed with supporting information and resources to facilitate the implementa- 
tion of the learning designs by teachers in areas beyond their immediate contexts 
(Figure 1). Within this database, quality learning designs are all characterised as 
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Figure 1. ICTs and their role in flexible learning. A repository of high-quality learning designs 
for e-learning (http://www.learningdesigns.uow.edu.au) (Australian University Teaching 

Committee, 2003) 


being forms of problem types derived from the work of Jonassen (2000). The 
learning designs are based on problem solutions of a rule-based, an incident- 
based, a strategy-based or a role-based form (Oliver et al., 2002). 

In choosing a benchmark of best practice for a learning design in e-learning, it is 
important to ensure that there is a deliberate form of learning experience supported 
and that the learning experience is well matched to the intended learning outcomes. 
Given the wide variation in learning outcomes being sought, subjects being covered 
and learning levels being considered, benchmarks for e-learning designs necessarily 
require considerable judgement and discernment. 


Learning resources for quality e-learning 

The learning resources in e-learning settings provide the content and course materials 
that underpin the conceptual change in the planned learning outcomes. In most e- 
learning settings there are substantial amounts of course material provided for learn- 
ers, and the factors of the materials that influence quality are in most instances the 
same as those that influence the quality of conventional resource sets. For example: 

• How well the resources support the planned learning. 

• The scope of the resources. 

• The currency of the resources. 

• Appropriateness of media usage. 

• The relevance of the resources. 

A growing awareness has emerged of the duplication and proliferation of online 
resources, and much has been written on the topic of reusability as both a design 
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and development strategy for online learning materials and as a general approach 
to the use of digital resources for teaching and learning (for example, Downes, 
2000). The reusability of learning resources offers many advantages to all stake- 
holders in the learning process and is now considered an important factor in any e- 
learning process. There are many benefits to be gained from such stakeholders, 
such as: 

• The administrative and financial agents who can benefit from the potential costs 
savings associated with reusing and sharing learning resources. 

• The policy-makers who are interested in the legal and ethical implications of copy- 
right and intellectual property about digital resources. 

• The instructional designers who can gain from design strategies that facilitate and 
support sharing and reuse. 

• The resource developers who can gain efficiencies and economies of scale from 
development strategies that ensure interoperability and a capability for use of 
resources beyond the context for which they are designed (for example, Downes, 
2000; Shepherd, 2000). 

The term learning object is commonly used today when referring to reusable digital 
learning resources. Many writers, however, find this term distracting and misleading, 
and are very cautious in its use (for example, Friesen, 2003). The IEEE Learning 
Technology Standards Committee (LTSC, 2005) describes a learning object as any 
entity digital or non-digital that can be used for learning education or training. Much 
of the current work with learning objects is seeking to provide systems and processes 
to enable teachers, when creating e-learning environments, to be able to discover and 
locate online resources that can be seamlessly incorporated into their learning 
settings. 

There has been a huge amount of work undertaken by a number of large organisa- 
tions and groups to facilitate the reusability and interoperability of digital learning 
resources; for example, the work of the IMS Global Learning Consortium, 
Advanced Distributed Learning, and IEEE. This work appears to be removing many 
of the barriers that have previously limited reuse of learning resources. The work 
being done to develop the Sharable Content Object Reference Model (SCORM) is a 
strong case in point. SCORM has been developed by the Advanced Distributed 
Learning Initiative and provides a design and development model for learning 
resources that strongly supports reusability and interoperability (Advanced Distrib- 
uted Learning Initiative, 2004). Discovery of stored resources is facilitated by the 
use of learning object metadata and shared vocabularies to provide descriptors for 
resources that can be used in the discovery process (for example, LOM, 2002; 
LTSC, 2005). 

There are a growing number of repositories and databases of digital resources built 
and developed to foster the reuse of learning resources. Popular virtual learning envi- 
ronments such as WebCT, Blackboard and Desire2Learn all support and encourage 
the development and use of resources that have been designed with reusability in 
mind, and these tools, plus many others, now provide functionality for the storage 
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Figure 2. The Flexible Learning Toolbox digital repository facilitating the discovery and reuse 
of learning resources (http://flexiblelearning.net.au/search.asp) (Flexible Learning Advisory 

Group, 2003) 


and retrieval of digital resources. Figure 2 shows the Flexible Learning Toolbox 
digital repository developed as part of an Australian project involving the large-scale 
development of e-learning resources for use in the vocational and educational sector. 
The project explored the design and development strategies needed to support the 
storage, discover and reuse of digital resources designed for specific settings. Many 
factors were found to influence the reusability of the resources and guidelines were 
prepared to guide developers and designers seeking these outcomes (Brownfield & 
Oliver, 2003). 

There are many factors that contribute to the quality of learning resources and 
among these a significant number are particular to the digital resources that support 
e-learning. In developing benchmarks for the purpose of ascertaining quality, it is 
important to ensure that the examples of best practice that are selected as the basis 
for the benchmarking process are reflective of the contemporary thinking in this area, 
especially in relation to reusability and interoperability; factors that many universities 
have yet to adequately deal with. 


Supporting quality e-learning 

The effectiveness of a student’s e-learning experience depends not only on learning 
design and learning resources, but also on the manner in which it is delivered and 
supported. Learning supports describe the measures and means by which learners 
interact with systems, peers, mentors and teachers in the learning process. Previ- 
ously, many thought that well-designed e-learning settings would facilitate indepen- 
dent learning with little or no need for interaction with others, in the same vein as 
distance education modes had traditionally operated. Experience and research has 
shown that this is not the case. Students in e-learning settings can derive many 
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benefits from well supported learning strategies. They frequently strive for the 
company of their co-learners and often they seek the support and involvement of a 
tutor to facilitate and guide their learning experiences. A number of writers argue 
quite strongly that effective learning settings must involve such forms of learner 
support as mentoring, modelling, coaching and scaffolding (for example, Dennen, 
2002). Providing these supports in e-learning, settings can be achieved through a 
variety of means and at many different levels, and has been the focus of considerable 
research in recent years (for example, Salmon, 2002). 

The provision of support mechanisms for students in e-learning settings contrib- 
utes to the learning experience in a number of ways: 

• It enables learners to establish a sense of belonging and involvement, a sense of 
community, which encourages and motivates participation; 

• Supports can scaffold learning and help students to undertake and complete activ- 
ities and tasks they might not be able to do on their own; 

• Learning supports in the form of communications and discussions provide oppor- 
tunities for higher-order thinking and conceptual development, often not evident 
in independent learning settings (for example, Brook & Oliver, 2004). 

The process of benchmarking best practice in the provision of support for e-learn- 
ing can be a movable feast. Research is continually discovering and demonstrating 
innovative technology-facilitated strategies and tools that provide learning opportuni- 
ties and enhancements over conventional forms. Figure 3 showcases MarkUp, an 
innovative tool that assists learners to make meaning from readings. Learners are able 
to post their thoughts and reflections, using ‘sticky notes’, into an online document 
and to view and share the postings made by others. When use of this tool was inves- 
tigated with learners, the act of marking-up readings in a deliberate fashion and 



Figure 3. MarkUp, an online tool facilitating reading comprehension and shared viewpoints 
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reviewing others’ comments was found to provide strong supports for reading 
comprehension and the development of learners’ metacognitive skills (McMahon & 
Oliver, 2003). 

There are now many strategies and tools that can be used to support e-learning. All 
virtual learning environments and courseware management systems provide facilities 
such as discussion boards, chatrooms and groupware as a matter of course. The 
quality issue revolves not around whether these tools are available, but the strategies 
and means teachers employ to make effective use of them in the learning setting. Best 
practice in e-learning and all forms of learning involves high levels of deliberate and 
planned learner support through these means. 


Concluding remarks 

The presented descriptions of examples of strong practice in e-learning provide some 
avenues for those looking to use benchmarking as a means of quality assurance. One 
of the difficulties benchmarking poses for quality assuring learning in higher educa- 
tion (for example, Achtemeir & Simpson, 2005) is the process of describing best prac- 
tice in a way that supports the necessary comparative process. There have been a 
number of recent attempts to describe and state standards as an alternative means. 
Standards provide levels of achievement of a benchmark that can be qualitatively or 
quantitatively measured. 

The use of standards provides a means for the quality and scope of services to 
be documented and for the provision of such to be monitored against stated 
objectives. While standards have been applied for many years to business services, 
it is only recently that they have been considered in education and training. 
McKinnon et al. (2000) provide a detailed discussion of standards as a means for 
quality assuring teaching and learning in higher education, and from this work we 
have developed a framework that we think can be used as a starting point for 
planning standards to assist the benchmarking process in quality assuring e-learn- 
ing (Figure 4). 

The teaching and learning framework described in Figure 4 is broad and includes 
many features that do not specifically relate to e-learning but describe issues needed 
to assure teaching and learning in general. In the framework, e-learning is important 
in both the inputs and processes areas, where the quality of the materials are consid- 
ered, as well as the provision of the learning experiences and the assessment of 
learning. In this paper, a number of quality issues have been discussed in relation to 
best-practice in e-learning, and these issues have been tied to a discussion of quality 
assurance processes involving benchmarking against these best practices. As more 
and more universities seek to use e-learning as a mode of delivery for their units and 
courses, and as more and more they are being held accountable for the quality of the 
services they provide, the need grows for accepted standards and benchmarks against 
which performance can be judged. 

This paper has discussed and described some of the main factors associated with 
quality performance in the provision of e-learning services in higher education and 
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Figure 4. A framework describing quality teaching and learning (Oliver et al., 2006) 


has provided examples of current practice that exemplify these. The paper has 
highlighted difficulties associated with quality assuring e-learning given the depth and 
breadth of the activity, but has demonstrated that there are guiding principles that can 
be used and there are examples of institutions seeking to explore benchmarking and 
standards as quality assurance processes for e-learning activities. 

As institutions across all sectors of education proceed to mainstream e-learning as 
an alternative form of programme delivery, the need for objective measures of qual- 
ity will grow. E-learning is an activity that has many costs and makes many 
demands. It will be important to be able to satisfy stakeholders that the activity is 
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providing the maximum possible return on investment and that further investment 
is warranted. To do this, there is a need to discover and document best practice 
models that institutions can use to grow their capabilities and performances, and 
also that benchmarks against quality can be demonstrated. 
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